Skip to content

feat: Search pipeline name in pipeline run API#129

Open
yuechao-qin wants to merge 1 commit intoycq/search-pipeline-run-legacy-filterfrom
ycq/search-pipeline-run-name
Open

feat: Search pipeline name in pipeline run API#129
yuechao-qin wants to merge 1 commit intoycq/search-pipeline-run-legacy-filterfrom
ycq/search-pipeline-run-name

Conversation

@yuechao-qin
Copy link
Collaborator

@yuechao-qin yuechao-qin commented Feb 26, 2026

TL;DR

Implemented search name in Pipeline Runs.

What changed?

Functional

  • API GET /api/pipeline_runs/

    • Search pipeline run name in filter_query. Example query (not URL encoded for example):
      /api?filter_query={"and": [{"value_equals": {"key": "system/pipeline_run.name", "value": "My pipeline run name"}}]}
  • API POST /api/pipeline_runs/

    • Copies the pipeline run name to the annotations table for future searching. Example pipeline run annotations table after API is called:
      pipeline_run_id key value
      abc123 system/pipeline_run.name My pipeline run name

Other

  • Implemented backfill_pipeline_name_annotations function to populate existing pipeline runs with name annotations

How to test?

uv run pytest tests/test_api_server_sql.py tests/test_filter_query_sql.py tests/test_database_ops.py
  • Create pipeline runs with various name configurations and verify that name annotations are properly created
  • Test the backfill functionality by removing annotations and running the backfill process
  • Verify that filter queries using the new SystemKey.NAME work with different predicate types (exists, equals, contains, in)
  • Confirm that both user annotations and system annotations coexist properly

Why make this change?

  • This enables users to search and filter pipeline runs by name.
  • By mirroring pipeline names as system annotations, pipeline names will be searchable via the annotations table.

@yuechao-qin yuechao-qin force-pushed the ycq/search-pipeline-run-legacy-filter branch from f8f6e26 to 004fc05 Compare February 27, 2026 21:55
@yuechao-qin yuechao-qin force-pushed the ycq/search-pipeline-run-name branch from f8b3f75 to 7a7a60b Compare February 27, 2026 21:55
@yuechao-qin yuechao-qin changed the base branch from ycq/search-pipeline-run-legacy-filter to graphite-base/129 February 28, 2026 10:24
@yuechao-qin yuechao-qin force-pushed the ycq/search-pipeline-run-name branch from 7a7a60b to 2b4e624 Compare February 28, 2026 10:25
@yuechao-qin yuechao-qin changed the base branch from graphite-base/129 to ycq/search-pipeline-run-legacy-filter February 28, 2026 10:25
@yuechao-qin yuechao-qin changed the base branch from ycq/search-pipeline-run-legacy-filter to graphite-base/129 March 1, 2026 03:22
@yuechao-qin yuechao-qin force-pushed the ycq/search-pipeline-run-name branch from 2b4e624 to 10cd01f Compare March 1, 2026 03:23
@yuechao-qin yuechao-qin changed the base branch from graphite-base/129 to ycq/search-pipeline-run-legacy-filter March 1, 2026 03:23

with session_factory() as session:
db_run = session.get(bts.PipelineRun, run.id)
db_run.extra_data = {"pipeline_name": ""}
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If string is empty "", trust that pipeline name is empty (no need to check in spec) and set pipeline name = "" in annotations table.

Comment on lines +301 to +303

_backfill_pipeline_names_from_extra_data(db_engine=db_engine)
_backfill_pipeline_names_from_component_spec(db_engine=db_engine)
Copy link
Collaborator Author

@yuechao-qin yuechao-qin Mar 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do these in a single transaction.

Also, look into removing phase 1 (extra_data) and see if we can do eveyrhting for phase 2 in a single query.

https://dev.mysql.com/doc/refman/8.4/en/json-search-functions.html

SELECT pipeline_run.id, "system/piepline_run.name", JSON_EXTRACT(execution_node.task_spec, ".componentRef.spec.name")
FROM pipeline_run
JOIN
execution_node

INSERT from SELECT

@yuechao-qin yuechao-qin changed the base branch from ycq/search-pipeline-run-legacy-filter to graphite-base/129 March 4, 2026 00:41
@yuechao-qin yuechao-qin force-pushed the ycq/search-pipeline-run-name branch from 10cd01f to cde71ec Compare March 4, 2026 00:42
@yuechao-qin yuechao-qin changed the base branch from graphite-base/129 to ycq/search-pipeline-run-legacy-filter March 4, 2026 00:42
@yuechao-qin yuechao-qin changed the base branch from ycq/search-pipeline-run-legacy-filter to graphite-base/129 March 4, 2026 04:57
@yuechao-qin yuechao-qin force-pushed the ycq/search-pipeline-run-name branch from cde71ec to d8e7a70 Compare March 4, 2026 04:59
@yuechao-qin yuechao-qin changed the base branch from graphite-base/129 to ycq/search-pipeline-run-legacy-filter March 4, 2026 04:59
@yuechao-qin yuechao-qin changed the base branch from ycq/search-pipeline-run-legacy-filter to graphite-base/129 March 4, 2026 05:06
@yuechao-qin yuechao-qin force-pushed the ycq/search-pipeline-run-name branch from d8e7a70 to 5169db1 Compare March 4, 2026 05:07
@yuechao-qin yuechao-qin changed the base branch from graphite-base/129 to ycq/search-pipeline-run-legacy-filter March 4, 2026 05:07
@yuechao-qin yuechao-qin changed the base branch from ycq/search-pipeline-run-legacy-filter to graphite-base/129 March 4, 2026 08:53
@yuechao-qin yuechao-qin force-pushed the ycq/search-pipeline-run-name branch from 5169db1 to 1907269 Compare March 4, 2026 08:53
@yuechao-qin yuechao-qin changed the base branch from graphite-base/129 to ycq/search-pipeline-run-legacy-filter March 4, 2026 08:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant